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REMARKS 

Applicant has carefully reviewed and considered the Office Action mailed on September 
26, 2000, and the references cited therewith. 

Claims 1, 3, 9, 12, 13, 14, 17, 18, 19 and 22 and new claims 26-31 are added; as a result, 
claims 1-31 are now pending in this application. Claims 1 and 1 1 are amended solely to correct a 
typographical error (which caused the 1 12 rejection) and not in response to any art reference or 
obviousness rejection by the Examiner, deleting an extraneous "second" descriptor. Claims 3 
and 12 are amended to independent form to include all limitations of their respective base and 
any intervening claim(s). 



Objections to the Drawings 

The drawings were objected to because at least reference numeral 1957 was not shown. 
Red-lined drawings and replacement sheets showing this and other corrections to the drawings 
Figs. 18A and 19A are attached hereto. No new matter has been added. Formal corrections will 
be made upon receipt of a Notice of Allowance. 

The Examiner objected to the drawings as not showing certain features specified in the 
claims. Applicant respectfully traverses the objection. Applicant notes that examples of a 
picker are picker 2000 of Figure 10 and 20 (see page 9 line 13; page 41 line 13; and elsewhere), 
however the scope of "picker" should not be limited to these examples. Applicant has amended 
claim 13 to replace "slide clamp" with - tray-transfer device --, as supported by an example 
(labeled 1800 and 1802) on page 68 and Figure 18. Applicant points to rotator 1960 and/or 
motor 1962 as examples of means for limiting of claim 14 (This is supported in the 
specification on page 67 lines 27 to page 68 line 2: 

"The rotator 1960 includes stops so that the trays will essentially be flipped through 180°. A 
rotator motor 1962 is used to drive the rotator. The rotator motor can also have stops or be a 
motor that works between 0° and 180°.") 

Applicant points to Figures 19C though 19G and page 70 lines 8-1 1 as drawing support for 
slider (e.g., elements 1928 and 1929): 

"In one embodiment, jaw 1930 is then slid laterally using slider motor 1928 and screw 1929, with 
a holder or pusher 1931 holding the devices in position above tray 89 as jaw 1930 is moved out of 
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the way." 

Thus, Applicant respectfully request that the objections to the drawings be withdrawn. 

§112 Rejection of the Claims 
Claims 1-25 were rejected under 35 USC § 1 12, second paragraph, as being indefinite for 
failing to particularly point out and distinctly claim the subject matter which Applicant regards as 
the invention. Applicant respectfully traverses the rejection as to the assertion of claims 3, 12, 
17, 20 and 22 requiring "simultaneously" since although embodiments are described in which the 
jaws are rotated simultaneously, the invention does not require that the jaws be rotated 
simultaneously in all instances (see Figures 19C-19G, for just one example). Similarly, example 
embodiments are described in which the trays engage one another, but this is not required. Thus, 
these claims are definite and clear without this change, and cover embodiments whether or not 
the jaws are rotated simultaneously. Applicant has also added new claims 27-31 that further 
distinguish embodiments that rotate the jaws simultaneously. In view of the amendments, 
Applicant respectfully requests that the §1 12 rejection of the claims be withdrawn. 



§103 Rejection of the Claims 

Claims 1, 2 and 1 1 were rejected under 35 USC § 103(a) as being unpatentable over 
Applicant's allegedly admitted prior art as discussed on pages 2-5 of the specification. Applicant 
respectfully traverses the rejection. The Examiner bears the initial burden of factually supporting 
any prima facie conclusion of obviousness. To establish a prima facie case of obviousness, three 
basic criteria must be met. First, there must be some suggestion or motivation, either in the 
references themselves, or in the knowledge generally available to one of ordinary skill in the art, 
to modify the reference or to combine reference teachings. Second, there must be a reasonable 
expectation of success. Finally, the prior art reference (or references when combined) must teach 
or suggest all of the claim limitations. M.P.E.P. §2142. 

Applicant respectfully submits that the Examiner had failed to meet his burden of 
providing a prima facie case of obviousness. Instead, a conclusory statement is provided in the 
Office Action that "Since the inverting operation is usually done by hand, the provision of a 
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mechanism to perform this function would have been obvious to a person having ordinary skill in 
the art seeking efficiency and reduced labor costs." Applicant respectfully requests that the 
Examiner provide a reference or showing that a person having ordinary skill in the art seeking 
efficiency and reduced labor costs would have been motivated to provide the combination of all 
of the limitations recited in claim 1 and 11, and would have had a reasonable expectation of 
success with such a combination. Thus, Applicant respectfully request that the rejections to these 
claims be withdrawn. 

Allowable Subject Matter 
Claims 3-10 and 12-25 were indicated to be allowable if rewritten to overcome the 
rejection(s) under 35 US C § 1 12 set forth in the Office Action. In view of the above-presented 
amendments, claims 3-10 and 12-25 are believed to be in condition for allowance. 

Unconside red Document 
The Examiner has noted on his Office Action Summary that the Yang article referred to 
on Applicant's Form 1449 filed November 19, 1999 was unreadable. A legible copy is attached 
hereto, along with a new Form 1449. Applicant respectfully requests that the Examiner 
review the Yang article and return the initialled Form 1449 with his next official 
communication. 




AMENDMENT AND RESPONSE UNDER 37 CFR § 1 .11 1 

Serial Number: 09/350,251 
Filing Date: July 8, 1999 

Title: TRAY FLIPPER AND METHOD FOR PARTS INSPECTION 



Page 12 

Dkt: 139.059US1 



Conclusion 

Applicant respectfully submits that the claims are in condition for allowance and 
notification to that effect is earnestly requested. The Examiner is invited to telephone 
Applicant's attorney (612-373-6949) to facilitate prosecution of this application. 

If necessary, please charge any additional fees or credit overpayment to Deposit Account 
No. 19-0743. 



Respectfully submitted, 
ARYE MALEKET AL. 
By their Representatives, 

SCHWEGMAN, LUNDBERG, WOESSNER & KLUTH, P.A. 
P.O. Box 2938 
Minneapolis, MN 55402 
(612) 373-6949 

Date By 

Charles A. Lemaire 
Reg. No. 36,198 

CERTIFICATR IJNDFR 37 CFR 1 .8: The undersigned hereby certifies that this correspondence is being deposited with the United States 
Postal Service with sufficient postage as first class mail, in an envelope addressed to: Commissioner of Patents, Washington, D.C. 20231 , on 
this 26th day of January 2001 . 

Charles A. Lemaire 

Name Signature 
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DETERMINATION OF THE IDENTITY, POSITION AND ORIENTATION 
OF THE TOPMOST OBJECT IN A PBLEs 
SOME FURTHER EXPERIMENTS 

H. S. Yang and A. C. Kak 

Robot Vision Lab 
School of Electrical Engineering 
Purdue University 
West Lafayette, IN 47907 



ABSTRACT 



In a recent report (7), we proposed algorithms for 
segmenting out the visible part of the topmost object 
from a pile of planar and curved objects. Planar objects 
considered were of convex polyhedral type, such as 
prisms, boxes, wedges, etc; and the curved objects were 
of the type that could be recognized uniquely by using 
the Extended Gaussian Image, such object-types being 
cylinders, cones, spheres, ellipsoids, toruses, etc. For 
input these algorithms used 3-D vision data acquired with 
a structured light scanner. 

Id this report, we will review some of the algorithms 
presented earlier and show our latest experimental results 
on scenes that are more complex than before. In our 
presentation at the conference, we also plan to show some 
manipulation experiments that for sensory feedback use 
3-D vision data analyzed according to the algorithms 
presented here. 



1. Introduction 

Seemingly simple operations like picking up the top- 
most object from a pile of objects can be exceedingly 
difficult for a robot, even when the robot is endowed with 
sensory capabilities for the purpose of dealing with a ran- 
dom environment. For many years, it was hoped by the 
computer vision community that we would succeed in 
implementing on a computer the laws of Gestalt organi- 
zation and stereo perception that allow us humans to per- 
form such tasks so effortlessly. More recently, the attitude 
has been that while research must continue in simulating 
these human abilities, which appear to be guided by 
knowledge and driven by expectations, we must in the 
meantime also look for purely engineering solutions to the 
sensory feedback problems of robots. 

During the past few years, many engineering solu- 
tions have been shown to be feasible. For example, if the 
aim is to simply pick objects from a bin and if the object 
surfaces are smooth it is possible to use a vacuum gripper 
[5]. If the. aim is a bit more sophisticated, such as sorting 
a pile of objects on the basis of shape, one must now use 



the sensory data (in most cases, vision) to not only locate 
the least occluded object — usually topmost -- but also 
recognize its shape in two or three dimensions; in addi- 
tion, it may be necessary to compute optimum holdsites. 

With vision sensing, because of difficulties with the 
segmentation and grouping of photometric information, it 
is now generally believed that when the objects involved 
are inherently 3-D in nature (as opposed to being 
flattish), one must resort to range maps for scene 
analysis. Of course, the ideal would be to use the 
human-like stereo vision capability for generating the 
range maps with two or more cameras; but, in practice, 
for reliable and sufficiently dense data, a more engineer- 
ing solution must be used — structured light. 

In [7], we presented algorithms for analyzing struc- 
tured light range maps for the visible part of the topmost 
object in a scene. Here, we will demonstrate more recent 
results on scenes that are more complex. We will also 
review the two algorithms: one for analyzing scenes con- 
sisting of planar objects and the other for analyzing 
scenes of curved objects; this we will do for the sake of 
completeness. 

The reader is also referred to [7] for a simpler 
approach to determining the center coordinates of a 
tesselated Gaussian spheres on which the measured sur- 
face normals are mapped; the purpose being to construct 
orientation histograms for object recognition.. 

The problem of locating and identifying the topmost 
object in a pile is a specialized case of scene analysis with 
3-D vision; much has been done in that area lately. 
Oshima and Shirai [6] have segmented scenes consisting 
of polyhedral and curved objects by using a region grow- 
ing technique; the overall scene was then described in 
terms of properties of regions and relations between 
them. Horn and Ikeuchi [4] have used the Extended 
Gaussian Image to determine the identity, orientation of 
an object that is a part of a small pile. Bolles [1] has 
used structured light range data to quickly and reliably 
locate cylinders of a specified diameter in a pile of 
cylinders. We have described in [2] a procedure for pile 
analysis that correctly segments a structured light gen- 
erated edge-vertex description of a scene consisting of 
convex polyhedral objects. Dessimoz et al. [3] have used 
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a matched filter type of implementation in which one first 
takes note of the geometry and the mechanics of how the 
end-effector would pick up an object; this consideration 
then leads to a set of features that can be invoked to 
identify the object if it is successfully grasped by the 
robot. 



2. Testing the Topmost Visible Surface for 
Planarity 

Before we can test the topmost surface for planarity, 
it must first be located. Fortunately, with 3-D vision data 
that is trivially accomplished by seeking out the pixel 
with the largest z-coordinate, which corresponds to the 
maximum height above the work table (Fig. lb). (If 
there exist many pixels whose maximal heights are ident- 
ical - this can happen if a straight edge of the topmost 
object is parallel to the work table - it is possible to 
choose any one of them as the topmost pixel.) An NxN 
set of pixels containing the topmost pixel is then defined 
as the topmost patch (Fig. lc); and a 3x3 set of patches, 
containing the topmost patch at the center, is then tested 
for the planarity of the topmost surface. The size of the 
patches (NxN) is heuristically determined; if it is too 
small, the planarity test may not be reliable. We have 
used N=8. 

The planarity test measures the variation in surface 
normals over this 3x3 set of patches; each surface normal 
being calculated by performing a least-mean-squares fit of 
a plane to the corresponding 8x8 cluster of pixels. To 
give greater credibility to the planarity test, we can also 
examine the the Gaussian and mean curvatures, since 
over regions that are declared to be planar both should 
be zero. (For further details on how we compute these 
two curvatures, the reader is referred to [8).) 




Fig. 1: (a) Depicted here is a piled scene in which a cub* 
is the topmost object, (b) Topmost point in this scene is 
a corner point of the cube, (c) Illustrated here is the top- 
most patch comprising 8x8 cluster of points where the 
topmost point is positioned at the center. 



If with these tests it is determined that the surface 
region under examination is indeed planar, the object (or 
at least its topmost surface) is declared planar; otherwise' 
it is assumed to be curved, In some cases it is possible 
that the 8x8 patch containing the topmost pixel might 
straddle an edge close by; clearly we do not wish such a 
patch to contribute to planarity decisions. These can be 
discarded from further consideration by putting a thres- 
hold on the error between the fitted plane and the pixel 
positions, as was done first in [6]. 



3. Segmentation or Visible Part of the Topmost 
Planar Object 

If the topmost visible surface is declared planar by 
the preceding procedure, the rest of this surface b 
extracted by a region growing procedure utilizing the 
simple heuristics that all the surface normals of the 
patches on the topmost surface must point in essentially 
the same direction (Surface Normal Constraint) and that 
the patches be adjacent (Adjacency Constraint). A 
region is allowed to grow until either there is nothing 
more to merge, or until we have exceeded some specified 
distance from the starting patch; this distance reflecting 
our knowledge of the maximum expected size of the 
objects in the scene. 

The process of merging can be made somewhat fas- 
ter if at the beginning we also estimate the directions 
toward which most of the topmost surface lies from the 
vantage point of the starting patch. For example, if we 
knew that the starting patch was at the top right corner 
of the surface, then the growing process could be confined 
to the left of and below the starting patch. 
Determination of where the starting patch lies in relation 
to the rest of surface can in some cases be made by exa- 
mining a neighborhood set of patches. 

After the topmost surface is extracted, the rest of 
the visible portion of the topmost object must also be 
segmented out. In a sense, this is a continuation of the 
region growing process mentioned above, and a similar 
set of heuristics is now called for; the ones that we use 
are: 

1. Adjacency Constraint: To act as a starting patch of 
an adjoint surface, a patch must be a neighbor of 
one of the patches of the extracted topmost surface. 

2. Surface Normal Constraints: Patches on an adjoining 
surface must have the same surface normals as the 
starting patch for that surface. In some cases, based 
on the knowledge available about the objects in the 
scene, constraints may be imposed on the permissible 
angles for the patch surface normals on adjoining 
surfaces. For example, if the objects are known to be 
rectangular, we may not accept a patch as a seed for 
an adjoining surface unless its surface normal is per- 
pendicular to the normal associated with the top- 
most surface. 
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3. Object Convexity Constraint: All surfaces that 
adjoin the topmost surface must also satisfy the con- 
vexity condition that says that those points that lie 
on a straight line connecting any location on the top- 
most surface and a point on an adjoining surface 
must all lie behind both surfaces when those points 
are viewed along the direction which is the inverse of 
the surface normal of either the topmost surface or 
the adjoining surface. As shown in Fig. 2, p a , which 
is on a line connecting Pi and P^ is behind St when it 
is viewed along v tl which is the inverse of the surface 
normal of the topmost surface; and behind S» when it 
is viewed along the V\. 

Fig. 3a illustrates a light stripe image of the scene in Fig. 
la, and Fig. 3b shows the topmost planar object (a cube) 
segmented out by the region growing procedure. 

Ak 




S^: topmost lurEtce 

S 4 : adjoining surface 

N,; surfac* wrmal at S v 

N t : suiTsc« DormaJ of S, 



Fig. 2: Illustrated here is a concave polyhedral object in 
which topmost surface (S t ) and its adjoining surface (S») 
satisfy convexity constraint. 




4. Isolation of the Topmost Curved Object 

Region growing is not the best strategy for extract- 
ing the visible portion of a topmost curved object; the 
difficulty being caused by variations in the surface nor- 
mals over such surfaces, 

A better strategy for curved objects is to locate the 
outer boundary of the topmost curved surface by finding 
the defining range discontinuities. This is accomplished in 
the following manner. From the topmost point, we 
sequentially proceed outwards in eight directions, E, W, 
M, S, NE, NW r SE, and SW, until along each direction 
we reach a point of a significant range discontinuity (Fig. 
1); this pursuit being abandoned if a discontinuity is not 
;*ound within a certain distance, which depends upon our 
,l priori knowledge of the maximum dimension of the 
objects in the scene. After the range discontinuity points 
.ire detected in the eight or fewer directions, we select one 
of those as a starting point for outer boundary tracking. 
The rest are designated as "posts" on which the 
extracted outer boundary must "bang". Suppose, after 
1 racking out from the starting point for a certain 
predetermined distance, a post is not encountered, we 
abandon that track, and select one of the other posts as a 
starting point. Tracking proceeds from post to post. After 
ime complete attempt at going around all the posts in one 
direction, we go back to those post pairs where tracking 
proved unsuccessful; for these the tracking is then 
*,ttempted in the opposite direction. If that also fails, we 
cither connect the two posts by a straight line; or if 
greater precision is demanded, we might set up another 
post between the offending pair and retry the procedure. 



sw 




Fig. 3: (a) Depicted here is a light stripe image of the 
scene in Fig. 1. (b) Illustrated here is the visible part of a 
cube at the top of the pile. 



TP: Topmoat Point 



Fig. 4: (a) Described here is the outer boundary of the 
topmost curved object and its posts in eight cardinal 
directions from the topmost point. 
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However, for an object tike a torus, whose range map 
from many viewpoints would contain a hole inside th*» 
outer boundary, some posts may not reside on the outer* 
boundary at all. As shown in Fig. 5, for the viewpoint 
shown there, post 5 is located on the inner boundary of i. 
torus, therefore the tracking process from post 4 to post 
5, and from post 5 to post 6, will fail even if we subdivide 
the sectors between post pairs. As a result, we could lose 
almost half of the topmost object surface if we simply 
connected posts 4 and 6. To overcome this flaw, whenever 
there exist post pairs where tracking fails, one must meti- 
culously examine regions around those posts to find out if 
there really exists a hole there, implying a torus like 
object. 

This examination can be carried out by checking the 
polarity of the Gaussian curvature of the points around 
those posts. (Negative Gaussian curvature indicates that 
the region around the point is locally saddle shaped.) If 
the existence of a hole around a post is verified, we must 
then try to locate the third range discontinuity along that 
direction ( p b in Fig. 5) and use it as a new post for the 
purpose of further tracking. 




this rectangle those points that arc within the object 
boundary, we have devised a procedure which uses ( De 
polygon filling technique of computer graphics 

The procedure is based on the principle that when a 
straight line intersects a closed boundary at, say, Pi ( p 2 

PS ,and Pn, all the points that lie on a chord between 

Pi and P2 must belong to the interior of the boundary* 
those that are on a chord joining P2 and P3 must be 
exterior to the boundary; those that are on a chord join, 
ing P3 and P4 must again be interior to the boundary* 
and so on (Fig. 8). The implication here is that if we can 
find all the horizontal scan lines that are simultaneously 
in the data frame and the extracted boundary of the top- 
most object, it should be a simple matter to fill out the 
interior. 

The determination of the intersections of scan lines 
with the boundary is aided by the fact that the computet 
representation of the boundary is that of a polygon. The 
points of intersection of a scan line with each side of the 
polygon can be found by the following simple routine. 

Let (I-A), and (I 2 ,J 2 ) be the two vertices of a particu- 
lar side of the polygon. We then calculate 

I2-I1 

If r < 0.0 or t > 1.0 , then the scan line does not touch the 
line segment; if r = 0.0 or 1.0, then the scanline intersects 
the start or ending vertices of the line segment, respec- 
tively. On the other hand, if 0.0 < r <l.o, then the scan- 
line intersects the interior of the line segment and the 
coordinates of the intersection point are given by 

J, = J, + r»(J 3 - J 0 

For each scanline, the intersection points with all 
sides of the polygon are collected, sorted from left to 
right in the order of increasing horizontal coordinate; and 
then the points between each pair are considered as the 
interior points of object region provided, of course, the 
range data is available at such points. 



Fig. 5: An example where the first range discontinuity to 
the south (P J from the topmost point on a torus does not 
reside on the outer boundary of the torus; if points 
around P» have negative Gaussian curvature, the third 
range discontinuity (P b ) to the south is supposed to be on 
the outer boundary of the torus. Tracking around P 5 is 
then retried. 




»ean!ine 



After outer boundary of the topmost curved object is 
extracted, we are faced with the not so simple task of 
extracting the interior points. 

This is accomplished by first extracting a minimum 
bounding rectangle containing the outer boundary points; 
this operation being trivial, but it does reduce the 
amount of data for further processing. To extract from 



Fig. 6: Illustrated here are the intersection points of a 
scan line with the object boundary approximated by a 
polygon. Points that lie on the chords between the 
Pi and P 2 , Pa and P<, P« and P 6 belong to the interior of the 
boundary. 



Fig. 7a illustrates a scene in which a cylinder is 
located at the top of the pile; Fig. 7b depicts a light 
stripe image of Fig. 7a; Fig. 7c shows the outer boundary 
of the cylinder; and Fig. 7d shows the visible part of the 
topmost object. 




5. Representation and Recognition of th 
Topmost Object 

For curved objects, after extracting the visible part 
we compute the local surface normals and surface curva- 
tires (the mean and the Gaussian curvatures) by the 
technique discussed in [8]; whereas for planar objects only 
ti e local surface normals are computed. Surface normals 
are then mapped onto a Gaussian sphere to construct a 
orientation histogram. In [7], we have presented what 
appears to be a particularly simple approach to the calcu- 
lation of the center coordinates of a Gaussian sphere. 

When the topmost object is planar, its identity and 
attitude are subsequently determined by matching the 
observed EGI with prototype EGFs stored in the 
rremory. For curved objects, in many cases the surface 
curvature information often provides direct clues as to 
the identity of the object; and in others, it definitely con- 
strains the object type involved. 

For instance, if the visible portion of the topmost 
curved object simultaneously contains points with nega- 
tive, zero and positive Gaussian curvatures, it can be 
identified as a torus; in that case, the attitude of the 
torus can be determined by calculating the local surface 
orientations of those points that are characterized by zero 
Gaussian curvature. Cylindrical, conical and elliptic sur- 
faces can also be discriminated by examining the polari- 
ties of Gaussian and mean curvatures and their distribu- 
tions. Although both cylindrical and conical surfaces have 
zi*o Gaussian curvature and negative mean curvature, a 
cylindrical surface can be differentiated from a conical 
surface since the mean curvatures of the points on the 
cylindrical surface are all identical, while those on the 
conical surface are not. Once the identity of the topmost 
curved object is determined using the surface curvature 
ir formation, its attitude is then determined by matching 
its observed EGI with its prototype. For further details 
a'xmt determining the identity and attitude of the top- 
rr.ost curved object, the reader is referred to [8]. 

At the conference, we plan to show manipulation 
experiments that demonstrate a robot first examining a 
scene with a 3-D vision sensor; calculating the identity, 
o-ientation and optimum gripping point of the topmost 
object in the pile; and then using this information to 
rr.ove the topmost object to a sorted pile consisting of 
similar objects. 



Fig. 7: (a) Illustrated is a piled scene in which a cylinder 
is located at the top of the heap, (b) Light stripe image 
of this scene, (c) Depicted here is the outer boundary of 
the cylinder, (d) Shown here is the visible part of the 
topmost object. 
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